From ShopTalk to ShopMobile: Vision-Based Barcode Scanning with Mobile Phones for Independent Blind Grocery Shopping

Independent grocery shopping is a major challenge for many visually impaired (VI) individuals [1]. In 2006, we began our work on ShopTalk, a wearable system for independent blind supermarket shopping [2]. ShopTalk consisted of a small OQO computer, a wireless barcode reader, and a numeric keypad, and was based on a simple insight: independent blind shopping = verbal route instrucKons + barcode scans. The system was the first aNempt reported in the accessible shopping literature to use shelf barcodes as topological points for locaKng products through verbal direcKons. In 2008-­‐09, we ported ShopTalk to a mobile phone plaRorm. This paper presents a vision-­‐based barcode scanning method that will allow ShopMobile, the next generaKon of ShopTalk, to run on a mobile phone with no external barcode scanner. KEYWORDS Accessible blind shopping, vision-­‐based barcode scanning, mobile compuKng BACKGROUND A typical modern supermarket stocks an average of 45,000 products and has a median store size of 4,529 square meters [3]. Many visually impaired (VI) people do not shop independently; they rely on friends, relaKves, volunteers, and store employees. When these individuals are unavailable, VI shoppers reschedule or postpone shopping trips. When they can reach the store independently, they experience frequent delays waiKng for store employees to assist them. Some staffers are unfamiliar with the store layout, others become irritated with long searches, and sKll others do not have adequate English skills to read the products' ingredients [4]. These difficulKes cause VI shoppers to abandon searching for desirable products or seNle for distant subsKtutes. PeaPod (hNp://www.peapod.com) and similar home delivery services provide grocery shopping alternaKves. However, such services are not universally available and, when available, require shoppers to schedule and wait for deliveries, thereby reducing personal independence and making spontaneous shopping impossible To help VI individuals overcome these challenges, in 2004 we began to develop RoboCart [5], a roboKc supermarket shopping assistant for VI shoppers. A long-­‐term collaboraKve agreement was negoKated with Lee's Market Place to grant us access to its store in Logan, UT, for RESNA Annual Conference – June 26 – 30, 2010 – Las Vegas, Nevada Making Assistive Technology and Rehabilitation Engineering a Sure Bet Copyright © 2010 RESNA 1700 N. Moore St., Suite 1540, Arlington, VA 22209-­‐1903 Phone: (703) 524-­‐6686 -­‐ Fax: (703) 524-­‐6630